Human Emotion Estimation Using Wavelet Transform and t-ROIs for Fusion of Visible Images and Thermal Image Sequences

Most studies in human emotion estimation focus on visible image-based analysis which is sensitive to illumination changes. Under uncontrolled operating conditions, estimation accuracy degrades significantly. In this paper, we integrate both visible images and thermal image sequences. First, to address limitations of thermal infrared (IR) images, such as being opaque to eyeglasses, we apply thermal Regions of Interest (t-ROIs) to sequences of thermal images. Then, wavelet transform is applied to visible images. Second, features are selected and fused from visible features and thermal features. Third, fusion decision using Principal Component Analysis (PCA), Eigen-space Method based on class-features (EMC), PCA-EMC is applied. Experiments on the Kotani Thermal Facial Emotion (KTFE) database show the effectiveness of proposed methods.